River Oemhii
   |
Posted - 2011.02.07 22:59:00 -
[1]
 Originally by: Cergorach
Good work Dev team CCP, very impressive improvements with relatively simple (sounding) solutions. I'm not very well at home in Python, you can shoot me with some missiles if I say something really stupid. But how difficult is it to move the calculations to a separate treads or cores? I understand that the that wouldn't speed up the individual missiles, but generally speaking folks don't shoot a single missile and often many missiles are in the air at one time. Would the syncing of the data be a problem that the version of Phyton you currently use doesn't provide for?
TL;DR: For Eve to move in to the Multithreaded World would require CCP to build a new Stackless Python Virtual Machine that does not use the GIL, no simple task. There may be better uses of staff time in the short-term?
I can give some insight on Python, threading, stackless python, and while we'll be stuck with the current system for a bit.
Some background, Python, is an interpreted language, meaning its code is not 'compiled' to machine assembler/byte code but read and dynamically executed by a Virtual Machine. In this case the most common interpretor is CPython.
Python's implementation requires using something called the GIL, Global Interpreter Lock. This prevents Python code from executing multiple threads at the same time. The reason is because some of CPython's methods and systems are not 'Thread Safe' meaning that if executed across multiple cores at once you may run in to issues such as race conditions that will kill your program.
At one point the authors of CPython did attempt to remove the GIL and setup context Locks on the methods that required it, this actually led to a decrease in performance and the changes were rolled back.
Here's a good presentation on the GIL: http://python.mirocommunity.org/video/1101/mindblowing-python-gil
Eve however, doesn't strictly use Python. It uses another implementation called Stackless Python. Stackless Python adds an important feature, mainly Green Threads. These are extremely lightweight threads that the program manages, versus the OS in most other cases.
Eve uses these extensively for object handling (I believe, fuzzy here, Dev?) and transactions. These Green Threads allow 1,000's, or 10,000's of threads to run at once. Each green thread (tasklet) is given a slice of cpu time to execute then yield. Returning back to the manager to let the next thread to run.
This is a Coroutine paradigm meaning that the entire state of the function stack is stored, so the function, even in the middle can be stopped, and yield back allowing the program to come back later and pick up execution where it left off.
Ok, so what does this have to with Eve and multicore/multithreading? Well there are a large number of Python implementations that do not use the GIL, for .Net/Mono: IronPython, Boo, JavaVM: Jython, but there are no alternate implementation of Stackless Python.
Unfortunately, one of the main useful features (Green Threads) of Stackless Python also makes it rather hard to create other Virtual Machines for.
For instance, .Net only supports limited Coroutines using Enumerables and the Yield keyword. There's no way in the framework to natively store the call/function stack and local values to restore the thread later. That means that IronPython/Boo would have to create System.Thread objects which are expensive and managed by the OS. The alternative is to re-write the entire tasklet/green thread system in the new platform: C/C++/.Net/Mono/Java etc.
So, for Eve to move in to the Multithreaded World would require CCP to build a new Stackless Python Virtual Machine that does not use the GIL, no simple task. |